PATENT 

5681-80400 

P9307 



"EXPRESS MAIL" MAILING LABEL 
NUMBER EL990142769US 
DATE OF DEPOSIT January 29. 2004 
I HEREBY CERTIFY THAT THIS PAPER OR 
FEE IS BEING DEPOSITED WITH THE 
UNITED STATES POSTAL SERVICE 
"EXPRESS MAIL POST OFFICE TO 
ADDRESSEE" SERVICE UNDER 37 C.F.R. 
§L10 ON THE DATE INDICATED ABOVE 
AND IS ADDRESSED TO THE 
COMMISSIONER FOR PATENTS, BOX 
PATENT APPLICATION, P.O. BOX 1450, 
Alexandria^VA 22313-1450 




Derrick Brown 



simultaneous execution of test suites on different platforms 



By: 

Victor Rosenman and Olga Kuturianu 



B. Noel Kivlin 

Meyertons, Hood, Kivlin, Kowert & Goetze) 
P.O. Box 398 
Austin, TX 78767-0398 



47859 



Ver. 47859S5.doc 



1 

Simultaneous Execution of Test Suites on Different 

Platforms 

CROSS-REFERENCE TO REIiATED APPLICATIONS 

'5 This Application relates to Application No. (STC File 

No. 47900), entitled ^'Automated Test Execution Framework with 
Central Management", and to Application No. (STC File 
No. 47979), entitled ^^Parallel Text Execution on Low-End Emula- 
tors and Devices". 

10 BACKGROUND OF THE INVENTION 

1 . Fxeld of 'the Inven'blon . 

[0001] This invention relates to improvements in soft- 

ware and hardware design verification. More particularly, this 
invention relates to methods and systems for centrally managing 
15 the composition and execution of test suites for verification 
of different hardware and software. 

2. Descrip-tlon of "the Related Ar-b. 

[0002] The meanings of acronyms and certain terminology 

used herein are given in Table 1: 

20 Table 1 



API 


Application programming interface 


CLDC 


Connected, limited device configuration. CLDC is 
suitable for devices with 16/32-bit RISC/CISC 
microprocessors/controllers, having as little 
as 160 KB of total memory available. 


HTTP 


HyperText Transfer Protocol 


ID 


Identifier 


IP 


Internet Protocol 


J2EE 


Java 2 Enterprise Edition 


J2ME 


Java 2 Micro Edition 


J2SE 


Java 2 Standard Edition 


JAD 


Java application descriptor 


JAR 


Java archive 


JDTS 


Java Device Test Suite execution framework 
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MIDlet 


A MIDP application 


MIDP 


Mobile information device profile. A set of Java 
APIs, which, together with the CLDC, provides a 
compxei-.e u^ixicj a,ppxxca ujion r u.riL.iiiie ciiv_Li.orinieriu 
targeted at mobile information devices. 


Plat- 
form 


An underlying system on which application 
programs can run on a computing device. It 
typically includes an operating system, and can 
include supporting hardware, which may be either 
standardized, or customized for a particular 
computing device. 



[0003] MIDP is defined in Mobile Information Device 

Profile (JSR-37}, JCP Specification, Java 2 Platform, Micro 
Edition, Ver 1.0a (Sun Microsystems Inc., Palo Alto, Calif or- 
5 nia, December 2000) . MIDP builds on the Connected Limited De- 
vice Configuration (CLDC) of the Java 2 Platform, Micro Edition 
(J2ME) (available from Sun Microsystems Inc., Palo Alto, Cali- 
fornia) . The terms Sun, Sun Microsystems, Java, J2EE, J2ME, 
J2SE, and the Sun logo are trademarks or registered trademarks 

10 of Sun Microsystems, Inc. in the United States of America and 
other countries. All other company and product names may be 
trademarks of their respective companies. A portion of the dis- 
closure of this patent document contains material that is sub- 
ject to copyright protection. The copyright owner has no objec- 

15 tion to the facsimile reproduction by any one of the patent 
document or the patent disclosure, as it appears in the Patent 
and Trademark Office patent file or records, but otherwise re- 
serves all copyright rights whatsoever. 

[0004] Tools have been developed . in recent years to aid 

20 in the design verification of hardware and software systems, 
for example software suites, hardware circuitry, and programma- 
ble logic designs. In order to assure that the design complies 
with its specifications, it is common to generate a large num- 
ber of input or instruction sequences to assure that the design 

25 operates as intended under a wide variety of circumstances. In 
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general, test systems produce a report indicating whether tests 
have been passed or failed, and, in some cases may even indi- 
cate a module that is estimated to be faulty. 

[0005] Conventionally, to test a device under develop- 

5 ment (such as a mobile information device) , or to test software 
designed to run on such a device, a developer connects the de- 
vice to an appropriate test system. The target device under 
test may be connected to the test system either directly or via 
a communication emulator. The developer selects a battery of 

10 test programs to run on the target device while monitoring its 
behavior. Running the complete battery of tests can commonly 
take many hours or even days. This problem is particularly 
acute in testing low-end computing devices, such as cellular 
telephones and other mobile information devices, which have 

15 limited computing power and memory resources. Thus, testing on 
the target device can become a serious bottleneck in the devel- 
opment cycle. 

SUMMARY OF THE INVENTION 

[0006] A centralized system for centrally managing test 

20 suites is disclosed in commonly assigned Application No. (STC 
File No. 47900), entitled ""^Automated Test Execution Framework 
with Central Management", which is herein incorporated by ref- 
erence. In this arrangement, a central repository contains a 
management unit, available test suites and a single test execu- 

25 tion framework, referred to herein as a test harness. Using the 
management unit, a system administrator establishes active ver- 
sions of the various test suites and their individual configu- 
rations. End users install clients of the central repository, 
using a system-provided installer program. In each client, an 

30 execution script is created, which downloads the test harness 
and a local configuration file. Then, when the test harness is 
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executed at the client, it loads with all designated test 
suites already installed, configured and ready for execution. 
The client always has the most current versions of all test 
suites. Advantageously, all necessary information is obtained 
5 from a single central location. 

[0007] A further improvement in test suite management 

is disclosed in commonly assigned Application No. (STC File 
No. 47979), entitled ""Parallel Text Execution on Low-End Emula- 
tors and Devices", which is herein incorporated by reference. 

10 While this arrangement facilitates testing large numbers of de- 
vices simultaneously, it do.es not address the problem of simul- 
taneously testing software, that is designed to execute on dif- 
ferent platforms. Nor does it address the related problem of 
simultaneously applying test suites to different computing de- 

15 vices, which employ different platforms. In order to avoid rep- 
lication of test suite development, test suites may be designed 
to execute on different platforms. However, each platform typi- 
cally provides a unique environment. Furthermore, the different 
platforms may require the use of different communication proto- 

20 cols and API's. Therefore, it would be desirable if test suites 
could be simultaneously executed on different platforms using a 
common test harness. 

[0008] Embodiments of the present invention provide 

methods and systems for parallel testing of multiple low-end 

25 computing devices (devices-under-test) , such as mobile informa- 
tion devices in which different test suites are executed simul- 
taneously on different platforms from a single instance of a 
test harness. The test harness has a component-based architec- 
ture. This, is exploited according to the instant invention by 

30 the substitution of various software components that support a 
test suite in order to execute the test suite on a new platform 
or on a. new computing device. For example, a platform-specific 
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API is provided for each of the platforms upon which the test 
suite is executed on the clients of the test harness, that is 
by the devices-under-test. Other platform-specific software 
components needed to support execution of tests, e.g., an exe- 
5 cution agent, and subsidiary classes, are implemented for each 
test suite according to the requirements of its respective 
platform-specific API. At runtime, the test harness deploys 
each test suite together with a compatible platform-specific 
execution agent and other platform-specific components. The 

10 agents execute the test suites, and communicate with the test 
harness, returning test results according to a known API. 

[0009] The invention provides a method for testing com- 

puting devices, which is carried out by providing a plurality 
of suites of test programs for access by a server, wherein a 

15 first suite and a second suite of the plurality of suites are 
respectively adapted to run on a first platform and a second 
platform. The method further includes storing a first execution 
agent that is adapted to run on the first platform and a second 
execution agent that is adapted to run on the second platform 

20 for access by the server, and coupling a first computing device 
and a second computing device of the computing devices to the 
server, wherein the first computing device is adapted to oper- 
ate using the first platform and the second computing device is 
adapted to operate using the second platform. The method fur- 

25 ther includes installing no more than one test harness on the 
server to support execution of the test programs by the. first 
computing device and the second computing device, using the 
test harness to package a first test object with the first exe- 
cution agent for download to the first computing device in a 

30 first package and to package a second test object with the sec- 
ond execution agent for download to the second computing device 
in a second package. The method further includes steps of re- 
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sponsively to an instruction of the test harness, downloading 
the first package and the second package to the first computing 
device and the second computing device, respectively, and con- 
currently executing a test program of the first package in the 
5 first computing device and a test program of the second package 
in the second computing device. 

[0010] According to an aspect of the method, the first 

suite and the second suite comprise platform-specific JAR 
files. 

10 [0011] According to yet another aspect of the method, 

the first package and the first package comprise JAR files. 

[0012] A further aspect of the method is carried out by 

displaying the suites as a hierarchy of identifiers of test ob- 
jects corresponding to the test programs, and selecting the 

15 first test object from' the first suite for execution thereof by 
the first computing device, and selecting the second test ob- 
ject from the second suite for execution thereof by the second 
computing device. 

[0013] The invention provides a computer software prod- 

20 uct, including a computer-readable medium in which computer 
program instructions are stored, which instructions, when read 
by a computer, cause the computer to perform a method for test- 
ing computing • devices, which is carried out by providing a plu- 
rality of suites of test programs for access by a server, 

25 wherein a first suite and a second suite of the plurality of 
suites are respectively adapted to run on a first platform and 
a second platform. The method further includes storing a first 
execution agent that is adapted to run on the first platform 
and a second execution agent that is adapted to run on the sec- 

30 ond platform for access by the server, and coupling a first 
computing device and a second computing device of the computing 
devices to the server, wherein the first computing device is 
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adapted to operate using the first platform and the second com- 
puting device is adapted to operate using the second platform. 
The method further includes installing no more than one test 
harness on the server to support execution of the test programs 
5 by the first computing device and the second computing device, 
using the test harness to package a first test object with the 
first execution agent for download to the first computing de- 
vice in a first package and to package a second test object 
with the second execution agent for download to the second com- 

10 puting device in a second package. The method further includes 
steps of responsively to an instruction of the test harness, 
downloading the first package and the second package to the 
first computing device and the second computing device, respec- 
tively, and concurrently executing a test program of the first 

15 package in the first computing. device and a test program of the 
second package in the second computing device. 

[0014] The invention provides a system for testing com- 

puting devices, including a communication interface for cou- 
pling a plurality of the computing devices thereto for use in 

20 communicating with the system via the communication interface, 
a memory, a single test harness object stored in the memory, a 
suite of test programs stored in the memory for execution by 
the computing devices that are coupled to the system, and a 
processor that accesses the suite and the test harness object, 

25 wherein the processor cooperates with the test harness object 
to download the test programs via the communication interface 
for execution by the computing devices coupled thereto, so that 
at least first and second computing devices among the plurality 
execute different first and second test programs from the 

30 suite. The system provides facilities to receive messages via 
the communication interface from the computing devices with re- 
spect to execution of the test programs, and to control the 
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execution of the test programs in the suite based on the mes- 
sages by communicating responses to the messages via the commu- 
nication interface, and wherein the first and second test pro- 
grams are adapted to respective first and second platforms, and 
5 the first and second computing devices operate using the first 
and second platforms, respectively. 

[0015] According to an aspect of the system, the first 

and second test programs are executed substantially simultane- 
ously under control of the processor. 

10 [0016] Still another aspect of the system the test har- 

ness object and the processor further cooperate to access first 
and second execution agents that are adapted to the first and 
second platforms, respectively, and to package the first and 
second test programs with the first and second execution 

15 agents, respectively, for download to the first and second com- 
puting devices. 

[0017] Another aspect of the system includes a graphi- 

cal user interface in the processor for displaying the test 
programs as a hierarchy for selection of the first and second 
20 test programs therefrom. 

[0018] According to one aspect of the system, the com- 

puting devices are coupled to the communication interface via a 
common test host. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 [0019] For a better understanding of the present inven- 

tion, reference is made to the detailed description of the in- 
vention, by way of example, which is to be read in conjunction 
with the following drawings, wherein like elements are given 
like reference numerals, and wherein: 

30 [0020] Fig. 1 is a block diagram that schematically il- 

lustrate systems for parallel testing of low-end computing de- 
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vices, in accordance with an embodiment of the present inven- 
tion; 

[0021] Fig. 2 is a block diagram that schematically il- 

lustrate systems for parallel testing of low-end computing de- 
5 vices, in accordance with an alternate embodiment of the pre- 
sent invention; 

[0022] Fig. 3 is a block diagram that schematically il- 

lustrates program components used in a test system, in accor- 
dance with an embodiment of the present invention; 
10 [0023] Fig. 4 is a detailed block diagram of aspects of 

the program components shown in Fig. 3, in accordance with an 
embodiment of the invention; and 

[0024] Fig. 5. is a flow chart illustrating a method for 

executing tests on a device that is attached to a test harness, 
15 in accordance with an embodiment of the invention. 

DETAII.ED DESCRIPTION OF THE INVENTION 

[0025] In the following description, numerous specific 

details are set forth in order to provide a thorough under- 
standing of the present invention. It will be apparent to one 

20 skilled in the art, however, that the present invention may be 
practiced without these specific details. In other instances 
well-known circuits, control logic, and the details of computer 
program instructions for conventional algorithms and processes 
have not been shown in detail in order not to unnecessarily ob- 

25 scure the present invention. 

[0026] Software programming code, which embodies as- 

pects of the present invention, is typically maintained in per- 
manent storage, such as a computer readable medium. In a cli- 
ent/server environment, such software programming code may be 

30 stored on a client or a server. The software programming code 
may be embodied on any of a variety of known media for use with 
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a data processing system. This includes, but is not limited to, 
magnetic and optical storage devices such as disk drives, mag- 
netic tape, compact discs (CD's), digital video discs (DVD's), 
and computer instruction signals embodied in a transmission me- 
5 dium with or without a carrier wave upon which the signals are 
modulated. For example, the transmission medium may include a 
communications network, such as the Internet. 

Sys-bem Overview. 

[0027] Reference is now made to Fig. 1, which is a 

10 block diagram that schematically illustrates a system 20 for 
parallel testing of multiple mobile information devices 24, in 
accordance with an embodiment of the present invention. The 
system 20 is built around a test server 22, which is described 
in greater detail hereinbelow. The devices 24 are client de- 

15 vices, and are typically low-end devices, with limited comput- 
ing power and memory, for example, cellular telephones or per- 
sonal digital assistants (PDA's). In the description that fol- 
lows, the devices 24 are assumed to comply with MIDP, but the 
principles of the present invention are equally applicable to 

20 other types of low-end computing devices, operating in accor- 
dance with other standards and specifications. The server 22 
typically comprises a programmable processor, and has suitable 
memory and communication interfaces, such as wireless or wired 
interfaces, for communicating with multiple devices 24 simulta- 

25 neously. 

[0028] Each of the devices 24 typically receives a 

unique identifier for communicating with the server 22, which 
can be assigned in accordance with the methods disclosed in the 
above-noted Application No. (STC File No. 47979) . 
30 [0029] Reference is now made to Fig. 2, which is a 

block diagram that schematically illustrates a system 30 for 
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parallel testing of multiple devices 24, in accordance with an- 
other embodiment of the present invention. In this embodiment, 
the server 22 communicates with the devices 24 through a test 
host 32, such as a personal computer or workstation. Multiple 
5 test hosts of this sort may be connected to the server 22 in 
parallel, but only a single host is shown in Fig, 2 for the 
sake of simplicity. The host 32 can simultaneously accommodate 
multiple devices 24. 

[0030] Reference is now made to Fig. 3, which is a 

10 block diagram that schematically illustrates software program 
components running on the server 22 and the devices 24, in ac- 
cordance with an embodiment of the present invention. Elements 
of this software may be provided to the server 22 and to the 
devices 24 on tangible media, such as optical or magnetic stor- 

15 age media or semiconductor memory chips. The software may be 
downloaded to the server 22, and alternatively or additionally, 
to the devices 24 in electronic form, for example, over a net- 
work or over the air. 

[0031] The server 22 comprises a test harness 40, which 

20 generates and deploys the tests to be carried out by the de- 
vices 24. The test harness 40 is realized as one or more ob- 
jects stored in the memory of the server 22. The test har- 
ness 4 0 may be implemented as a modification of the ^'Java De- 
vice Test Suite" execution framework (JDTS) (version 1.0 or 

25 higher), available from Sun Microsystems, Inc., which employs 
MIDP. The test harness described in the above-noted Applica- 
tion No. (STC File No. 47900) can be modified for use as the 
test harness 40 by the application of ordinary skill. 

[0032] A test manager 42 in the server 22 is responsi- 

30 ble for serving requests from the devices 24, based on the 
unique client identifiers mentioned above. Typically, whenever 
one of the devices 24 makes a request, the test manager 42, 

P9308 



47859 



Ver. 47859S5.doc 



12 

typically operating as a main thread, reads the request and as- 
signs a new thread 44 to handle it. This thread 44 retrieves 
the client unique identifier from the request, calls the compo- 
nents of the test harness 40 that are needed to process the re- 
5 quest, and then returns the appropriate response to the client 
device, as described hereinbelow. After assigning the thread 44 
to handle the client, the main thread of the test manager 42 
waits for the next client request. Each client request is han- 
dled by a separate thread 44, which terminates upon completion 
10 of processing. This arrangement, together with the unique iden- 
tifier mechanism, ensures that the server 22 is able to handle 
multiple devices 24 simultaneously without confusion. 

[0033]- In order to run Java applications, the de- 

vices 24 contain an implementation of the Connected Limited De- 
15 vice Configuration specification, CLDC 46, with an implementa- 
tion of the Mobile Information Device Profile specification, 
MIDP 48, running over the CLDC 46. The applications that run on 
this technology, such as the tests supplied by test harness 40, 
are known as MIDlets. These applications are created by extend- 
20 ing an API MIDlet class of the MIDP 48. Thus, each test bun- 
dle 52 is actually a MIDlet, packaged in the form of a JAD/JAR 
file pair. At least a portion of the test bundle 52 is typi- 
cally downloaded to the devices 24 in a two-step process: 
[0034] 1. The server 22 downloads the JAD 

25 file, which contains environment settings and some 

environment demands. Application Manager Software, 
AMS 50, which is typically a part of a browser built 
into the devices 24, evaluates the JAD file to ensure 
that the device is able to accept the MIDlet. For ex- 
30 ample, the JAD file for a given MIDlet may specify 

that the device must support MIDP version 2.0. If the 
device does not support this version, the AMS 50 will 
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reject the application download, and will save the 
time that would otherwise be consumed by downloading 
the much larger JAR file. 

[0035] 2. After completing all the relevant 

5 checks, the AMS 50 reads from the JAD file the loca- 

tion of the corresponding JAR file on the server 22 
and asks to download the JAR file to one or more of 
the devices 24. The JAR file contains all the rele- 
vant classes of the test bundle 52. 
10 [0036] Once the JAR file for the relevant portion of 

the test bundle 52 is downloaded to one of the devices 24 and 
stored in the local device memory, the device is ready to run 
the tests of the test bundle 52. Every JAR file that the AMS 50 
downloads to the devices 24 typically contains an agent 54, 
15 which is used to run the tests, in addition to classes corre- 
sponding to the tests themselves. To start test execution the 
AMS 50 runs the agent class. The agent 54 then addresses the 
server 22 in order to receive instructions regarding the next 
test to run (getNextTest ) and to report test results (sendTest- 
20 Result), typically using a protocol based on HTTP. Each test in 
the test bundle 52 corresponds to a respective class in the JAR 
file. Each client request that is addressed by the agent 54 to 
the server 22 includes the unique identifier that has been as- 
signed to the particular one of the devices 24, so that the 
25 server 22 is able to recognize the client and serve it in the 
correct manner. Further details of the implementation of the 
.server 22 are given in the above-noted Application No. (STC 
File No. 47979). 

[0037] The test harness 4 0 has a component based archi- 

30 tecture. An API is provided for independent components of the 
tests for each of the platforms upon which the test bundle 52 
is executed at the devices 24. At the test harness 40, plat- 
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f orm-specif ic components are implemented for each test bun- 
dle 52 according to the respective platform-specific API. At 
runtime, the test harness 40 deploys each test bundle 52 to- 
gether with the agent 54, which is platform-specific, and con- 
5 figured according to the components of the particular test bun- 
dle 52. In each of the devices 24, the agent 54 executes the 
test bundle 52, and communicates with the test harness 40, re- 
turning test results according to a common API. 

Tes'b Sui-te Selecbxon. 

10 [0038] Reference is now made to Fig. 4, which is a de- 

tailed block diagram of the test harness 40 (Fig. 3) in accor- 
dance with a disclosed embodiment of the invention. Each of the 
available test suites 42 is associated with one of a group of 
execution agent JAR files 44 and one of a group of test API JAR 

15 files 46. The agent JAR files 44 and the API JAR files 46 are 
platform-specific. They are accessed by the test harness 40 
when packaging tests for download. The tests typically are 
packaged in the form of Java applications contained in a set of 
JAD and JAR files. Each JAR file of this sort, together with 

20 its accompanying JAD file, is referred to hereinbelow as a test 
bundle 52. As explained in further detail hereinbelow, users of 
the system 20 (Fig. 1) or the system 30 (Fig. 2) interact with 
the test harness 40 in order to select the tests to be executed 
by the system. Alternatively, other test harnesses may be used 

25 for generating the required test files, as will be apparent to 
those skilled in the art. 

[0039] The test harness 40 is provided with a user in- 

terface 48 that permits visualization of all available test 
suites and selection of tests of the test suites to be run. As 

30 is explained in further detail hereinbelow, the user inter- 
face 48 displays the test suites and the tests comprising the 
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test suites as a hierarchy of test names or identifiers that 
reference test program objects. 

[0040] Using a session component 50^ which stores all 

the test suites, a tree is constructed by the test harness 40 
5 that contains all the test suites. With the user interface 48 
the tree is displayed for the user, who can choose any combina- 
tion of tests from the display. The user selections are trans- 
mitted to the session component 50. The session component 50 
relates to a test suites manager 52, which further interoper- 
10 ates with the test harness 40 as explained below. 

Test: Sul'be Deployment:. 

[0041] The test harness 40 includes a deployer 54, 

which receives tests from the test suites manager 52 that have 
been selected by the session component 50 from the test 

15 suites 42 for packaging into JAR files. In the deployer 54 each 
test is associated with one of test suites 42 that are held in 
the session component 50. The deployer 54 accesses the corre- 
sponding" one of the agent JAR files 4 4 and the API JAR files 4 6 
and creates a new JAR file that includes the test itself and 

20 its associated specific execution agent and test API. As noted 
above, one or more these new JAR files form a test bundle 52, 
which is downloaded to the devices 24 or to a separate test 
host, for example the test host 32 (Fig. 2). Alternatively, the 
download could be directed to an emulator (not shown) of the 

25 devices 24. An executor 56 initiates the download of the new 
JAR files to the devices 24. 

A code fragment shown in Listing 1 presents further details 
of the current software implementation of the deployer 54. As 
explained above, the devices 24 are able to execute the tests 

30 as they employ the agent classes of the particular test bun- 
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die 52, which are specific to the test suite and compatible 
with the different platforms being used by the devices 24. 

[0042] Reference is now made to Fig. 5, which is a flow 

chart illustrating a method for executing tests on a device 
5 that is attached to a test harness in accordance with a dis- 
closed embodiment of the invention. Only one device is illus- 
trated for clarity of presentation, but it will be understood 
that different devices, operating on different platforms, can 
execute tests concurrently. The process steps are shown in a 
10 particular sequence in Fig. 5. However, it will be evident to 
those skilled in the art that many of them can be performed in 
parallel, asynchronously, or in different orders. 

[0043] The process begins at initial step 58. Here a 

group of tests is selected for execution on a device-under- 
15 test. The device-under-test is connected to a test harness. 

[0044] Next, at step 60, a current test from the group 

defined in initial step 58 is chosen. 

[0045] Next, at step 62 a pre-defined test suite that 

includes the current test chosen at step 60 is associated with 
20 the current test. 

[0046] Next, at step 64 an object is retrieved from a 

store corresponding to the test suite identified at step 62. 

[0047] Next, at step 66 JAR files are extracted from 

the object that was retrieved in step 64. The JAR files are 
25 specific to the platform of the device-under-test. 

[0048] Next, at step 68 classes required t^o operate the 

current test are extracted from the JAR files t^hat were ob- 
tained in step 66. These classes include an execution agent, 
corresponding to the agent 54 (Fig. 3) . 
30 [0049] Next, at step 70 a new JAR file and an accompa- 

nying JAD file are packaged. The JAR file includes the classes 
extracted at step 68, including the execution agent . 
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[0050] Next, at step 72, the new JAD and JAR files that 

were prepared in step 7 0 are downloaded to the device-under- 
test for execution. 

[0051] Control now proceeds to decision step 74, where 

5 it is determined if more tests remain to be processed. If the 
determination at decision step 74 is affirmative, then control 
returns to step 60. 

[0052] If the determination at decision step 74 is 

negative, then control proceeds to final step 76 and the proc- 
10 ess terminates. However, the process may be immediately re- 
peated so long as there are additional devices available for 
connection to the test harness. Such devices may operate on the 
same or on different platform as the first device. Thus, using 
the method illustrated in Fig. 5, any number of different de- 
15 vices may execute tests simultaneously, on different platforms, 
using only a single test harness. 

[0053] It will be appreciated by persons skilled in the 

art that the present invention is not limited to what has been 
particularly shown and described hereinabove. Rather, the scope 
20 of the present invention includes both combinations and sub- 
combinations of the various features described hereinabove, as 
well as variations and modifications thereof that are not in 
the prior art, which would occur to persons skilled in the art 
upon reading the foregoing description. 

25 

COMPUTER PROGRAM LISTINGS 



Listing 1 



30 public class DeployerThread extends Thread { 
/* Thread is a standard Java' Class*/ 

private boolean deployerActive = false; 
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protected Iterator recordslterator = null; 
protected JarMaker jarMaker = null; 
protected BundleDispatcher dispatcher = null; 
protected String sep = 
5 System . get Property ( " f ile . separator" ) ; 

protected Hashtable hashOf InnerClasses = new Hashtable(); 
protected TestSuitesManager tsm = null; 
protected String currentTS = " " ; 
protected TestSuite testSuite = null; 
10 private Default JarDeployer jarDeployer; 
protected String delimeter = null; 
private boolean flag = false; 
private Environment props; 

private FrameworkEnvironmentConf iguration conf ig; 

15 

/** 

* this method gets list of agent (platform- specif ic) 

* jars required for the test suite, which is 

* currently executed 
20 */ 

protected Vector getAgent JarsList ( ) { 
//get controlling properties 
boolean deployAgentClasses = true; 
String deployAgentClassesString = 
25 verif yProperty ( "DeployAgentClasses" ) ; 

if ((deployAgentClassesString != null) && 
deployAgentClassesString . toLowerCase( ) . equals ( "false" ) ) 
deployAgentClasses = false; 
30 //get platform- specif ic jars 

else if (deployAgentClasses) { 

appendJar ( "Agent Jar " , agent Jars ) ; 
appendJar ( "TestAPI Jar " , agent Jars) ; 

} 

35 

return agent Jars; 
} 



40 • /** 

* Patch the deployer props that had JarMaker according to 

* the current test suite 
*/ 

protected Properties getDeployerProps ( ) { 
45 Properties deploymentProps = j arMaker . getDeployerProps () ; 
String agentProps = 

deploymentProps . getProperty ( " agentprops " ) ; 

if (testSuite !=null) { 
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String agentJar = 

testSuite . getTestSuiteProperty ( "AgentJar" ) ; 

if ( agentJar ! =null ) { 
5 Properties defaultProps = new Properties () ; 

try 
{ 

URL[] url = { new File ( agent Jar ). toURL ()} ; 
URLClassLoader loader = new URLClassLoader ( url ) ; 
10 InputStream is = 

loader . getResourceAsStream( "agent classes . map" ) ; 

if (is !=null) { 

defaultProps . load( is ) ; 

15 } 
} 

catch ( Exception e ) 
{ 

e . getMessage ( ) ; 
20 e .printStackTrace ( System .err) ; 

} 

Enumeration keys = defaultProps . propertyNames () ; 

25 while ( keys . hasMoreElements () ) { 

String name = (( String) keys . nextElement ()). trim( ) ; 
String value = defaultProps . get Property (name ) ; 

if (name . equals ( "RemoteCommunicator " ) ) { 
30 agent Props = checkAndAddProperty ( agentProps , 

"communicator" , value) ; 

} 

else if ( name . equals ( "TestLauncher " ) ) { 

agent Props = checkAndAddProperty ( agentProps , 
35 "TestLauncher", value); 

} 

else { 

throw new IllegalArgumentException 

( "agentclasses . map contains unexpected property. \n' 
40 + 

"should define communicator and launcher only"); 

} 



45 } 



deploymentProps . setProperty ( " agentprops " , agentProps ) ; 
} 
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return deploymentProps ; 
} 

protected Properties getAgentProps 
5 ( /*Properties deploymentProps , Properties modeProps*/ ) { 

Properties deploymentProps = 

j arDeployer . getDeploymentProps ( ) ; 

Properties modeProps = props . getModeProps () ; 
10 String agentNames = 

conf ig . getFrameworkProperty ( ''AgentProperties" ) ; 
StringTokenizer st = new StringTokenizer ( agentNames ; 
String agentProps = 

15 while ( St . hasMoreTokens () ) { 

String propertyName = (( String ) st . nextToken ()). trim () ; 
String name = 

propertyName . substring ( 0 , propertyName . indexOf ( ) + 1 ) ; 

20 agentProps += name + 

deploymentProps . getProperty 

(propertyName . substring (propertyName . indexOf ( ) 4- 1 , 
propertyName . lengthC )) ) + "\n"; 

} 

25 

if (agentProps . endsWith( " \n" ) ) 

agentProps = 

agentProps . substring ( 0 , agentProps . lastlndexOf ( " \n" ) ) ; 

30 

if ( Iprops . getExecutionMode ( ) . equals ( "Standard" ) ) { 
Enumeration keys = modeProps . propertyNames () ; 
while ( keys . hasMoreElements ( ) ) { 

String key = ( String ) keys . nextElement ( ) 
35 agentProps +="\n" + key + "=" + 

modeProps . getProperty ( key ) ; 

} 

if (agentProps . endsWith( " \n" ) ) 
40 agentProps = 

agentProps , substring ( 0 , agentProps . lastlndexOf ( " \n" ) ) ; 

} 

if (props . getExecutionMode ( ) . equals ( "Standard" ) && 
45 ! props . getSubMode ( ) . equals ( " " ) ) 

agentProps+=" \nScreenAnalyzer=" + 
deploymentProps . getProperty ( "ScreenAnalyzer" ) ; 
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agentProps+=" \nTestBeansServerURL=http : //" + 
verif yProperty ( "ServerName" ) + + 
verif yPropertyC "ServerPort" ) + "/test"; 

5 if (test Suite !=null) { 

ComponentManager . getComponent 
( "TestManager " ) ) . getTestSuitesManager ( ) ; 
String agentJar = 

testSuite . getTestSuiteProperty ( "AgentJar" ) ; 
10 if ( agent Jar ! =null ) { 

Properties defaultProps = new Properties () ; 
try{ 

URL[] url = { new File ( agent Jar ), toURL ()} ; 
URLClassLoader loader = 
15 new URLClassLoader ( url , null); 

InputStream is = 

loader . getResourceAsStream( "agentclasses .map" ) ; 
if (is !=null) { 
20 defaultProps . load( is ) ; 

} 

} 

catch(Exception e) { 
e . getMessage ( ) ; 
25 e . printStackTrace ( System . err ) ; 

} 

Enumeration keys = defaultProps . propertyNames ( ) ; 
while ( keys . hasMoreElements ( ) ) { 
30 String name = 

(( String) keys . nextElement ( ) ) . trim ( ) ; 
String value = defaultProps . getProperty ( name ) ; 
if ( name . equals ( "RemoteCommunicator " ) ) { 

agentProps = checkAndAddProperty ( agentProps , 
35 "communicator", value); 

} 

else if ( name . equals ( "Test Launcher" ) ) { 

agentProps = checkAndAddProperty ( agentProps , 
"TestLauncher " , value) ; 
40 } ■ 

else { 

throw new IllegalArgumentException 

( "agentclasses . map contains unexpected property. 
\n" + 

45 "should define communicator and launcher only"); 

} 

} 
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} 

return deploymentProps ; 
} 

5 

protected void appendJar 

(String newJarName, Vector jars) { 
String newJar = 

testSuite . getTestSuiteProperty ( new JarName ) ; 

10 

i f ( new Ja r==nu 11) 

new Jar = props . getProperty ( new JarName ) ; 

if ((newJar == null) II ( ! newJar . endsWith ( " . j ar " ) ) ) 
15 System . out . println ( new JarName + 

■ " will not be included into deployed jars." + 
" If romized - IGNORE"); 
else { 

newJar = newJar . trim{ ) ; 

20 

if (! jars . contains (newJar ) ) 
j ar s . add ( newJar ) ; 

} 
} 

25 

/** Verifies correctness of the required property 
* Can be used by extending classes 
*/ 

protected String verifyProperty ( String propName) { 
30 String propValue = props , getProperty (propName ) ; 

if ((propValue == null) II (propValue . length ( ) == 0)) { 
/* if (propName . equals ( "ServerName" ) ) { 
try{ 

return InetAddress . getLocalHost ( ) . getHostName ( ) ; 
35 } catch (UnknownHostExcept ion uhe) { 

uhe . printStackTrace ( ) ; 

} 

}*/ 

40 System . out . println (propName + " property not defined"); 
return null; 

} 

return propValue . trim( ) ; 

} 

45 

/** Does the deployment work 
*/ 

public void run() { 
deployerActive = true; 
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try { 

//prepare deploy ement 

j arMaker . prepareDeployment ( ) ; 

5 try { 

//list tests for deployment 

while ( deployerActive && recordsl terator . hasNext ( ) ) { 
TestRecord testRecord = (TestRecord) 
10 recordslterator . next ( ) ; 

//no more TestRecords available - terminate 
// deployment 
if ( testRecord==null ) { 
deployerActive=f alse ; 
15 return; 
} 



/ /Check test suite 
String tsName = 
20 testRecord . getValue ( testRecord . TestSuite ) ; 

if (tsName == null) 

throw new NullPointerException 
("No test suite name provided to the Deployer"); 

25 if ( I currentTS . equals (tsName) ) { 

//test suite changed 

//update test suite specific values 
currentTS = tsName; 

testSuite = tsm . getTestSuite ( tsName ) ; 

30 

//j arMaker should be updated with thw agent jars 

//according to the current testsuite 

j arMaker . setAgent Jars ( getAgent JarsList ( ) ) ; 

35 //it also should be updated with new properties 

//according to the current test suite 
// jarMaker . setDeployerProps ( getDeployerProps ( ) ) ; 
j arMaker . setDeployerProps ( getAgentProps ( ) ) ; 
f lag=true ; 

40 

if (testSuite == null) 

throw new NullPointerException 
("There is no TestSuite object for the test suite: " 
+ tsName ) ; 

45 //close jars and start continue 

// deployement from a new jar 
//this is to allow only 1 test suite in jar 
j arMaker . terminateDeployment ( ) ; 
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//closesand deploys currently open jar 

j arMaker . prepareDeployment ( ) ; 

//marks a start for the further deployment 

} 

5 

//dispatch test record 

TestEntry te = dispatch ( testRecord, testSuite) ; 

//deploy jar files 
10 Maker . deployEntry( te , testSuite) ; 

} 

} catch ( ConcurrentModif icationException e) { 
System . out . println 

( "Deployment abnormally terminated" ) ; 
15 e . getMessage ( ) ; 

e . print St ackTrace ( ) ; 
deployerActive = false; 
return; 

} 

20 

//terminate deployment 

j arMaker . terminateDeployment ( ) ; 

System . gc ( ) ; 

System . out . println (" free memory: " + 
25 Runtime . getRuntime ( ) ..f reeMemory ( ) ) ; 

System . out . println ( "used memory: " + 
( Runtime . getRuntime ( ) . totalMemory ( ) - 
Runtime . getRuntime ( ) . f reeMemory ( ) ) ) ; 

} 

30 catch (lOException ie) { 

ie . printStackTrace ( ) ; 

System . out . println ( "Deployment failed" ) ; 
return; 

} 

35 

//test deployment completed 
dispatcher . deploymentCompleted ( ) ; 

//add notification to the server, so in serial mode, 
//the server could start the getNextApp request . 
40 j arDeployer . notif yServerListener ( ) ; 

System . out . println ( " Deployment completed" ) ; 
deployerActive = false; 

} 

45 /** Creates TestEntry corresponding to this test record 
* and appends it to the vector of test entries 
*/ 

protected TestEntry dispatch ( TestRecord testRecord, 
TestSuite ts) { 
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//list all files required to be deployed 
ArrayList allFilesList = new ArrayList(); 
list All Files (allFilesList, testRecord, 
ts . getTestSuiteProperty ( " TestClassesDir " ) ) ; 

5 

// creating TestEntry and prepare jar files list for 
// deployment 

String[] allFiles = new String [ allFilesList . size ()] ; 
Iterator iter = allFilesList . iterator () ; 
10 for(int i = 0; iter . hasNext ( ) ; i++) 
{ 

allFiles[i] = (String) iter.next(); 
if(allFiles[i] . startsWith ( File . separator ) ) 
allFiles[i]=allFiles[i] . 
15 substringC 1 , allFiles [ i ] . length( ) ) ; 

} 

allFilesList = null; 
//update vector of TestEntries 
20 String testClass = 

testRecord . getValue ( TestRecord . TestPackage ) . 

replace ('/'/'.')• replace ( ' \ \ ' . ' . ' ) 

+ " . " + testRecord . getValue ( TestRecord . TestClass ) ; 

25 if (testClass . startsWith ( " . " ) ) 
testClass = 
testClass . substringC 1 , testClass . length ( ) ) ; 

//add test suite properties 
30 Properties tsProps = 

testSuite . getTestSuiteProperties ( ) ; 
Properties testProps = 
getPropert ies ( testRecord . getValue 
( TestRecord . TestProperties ) ) ; 
35 testProps . put All ( ts Props ) ; 

//Remove jad and manifest properties from test 
//properties list 

//The following syntax expected: 
//<jad> . 1=JAD PROPERTY 
40 //<jad> . 2=JAD PROPERTY 

//etc. 

Vector j adPropsVector = new Vector(); 
Vector manif estPropsVector = new Vector(); 
for (Enumeration e = testProps . propertyNames () ; 
45 e . hasMoreElements ( ) ;) { 

String tp = (String) e . nextElement ( ) ; 
if ( tp . startsWith( "<jad> . " ) ) { 

j adPropsVector . add ( testProps . get Property ( tp ) ) ; 
//add the property value to the list of jad 
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// properties 
testProps . remove ( tp ) ; 

//remove jad property from the test properties 
//list 

5 } 

else if ( tp . startsWith( "<manifest> . " ) ) { 
manif estPropsVector . add 

( testProps . getProperty ( tp ) ) ; 
//add the property value to the list of jar 
10 //manifest properties 

testProps . remove ( tp ) ; 

//remove jad property from the test properties 
//list 

} 

15 } 

//this test is needed since jadProps has to be null if 
// empty (according to internal convention) 
String[] jadProps = null; 

20 

if ( ! j adPropsVector . isEmpty ( ) ) { 
int size = j adPropsVector . size () ; 
jadProps = new String [ size ] ; 
for (int i = 0; i < size; i-h-f) 
25 jadProps[i] = (String) j adPropsVector . elementAt ( i ) ; 

} 

//this test is needed since manif estProps has to be 
// null if empty (according to internal convention) 
30 String[ ] manif estProps = null; 

if ( ! manif estPropsVector . isEmpty ( ) ) { 
int size = manif estPropsVector . size () ; 
manif estProps = new String[ size] ; 
for (int i = 0; i < size; i-f+) 
35 manif estProps [ i ] = (String) 

manif estPropsVector . elementAt ( i ) ; 

} 

String propsString = constructPropsString( testProps ) ; 
40 return new 

TestEntry ( testRecord . getValue ( TestRecord . TestName ) , 
testClass, propsString, allFiles, jadProps, 
manif estProps ) ; 

} 

45 

private Properties getProperties ( String testProps) { 
Properties props = new Properties () ; 
if ( testProps==null ) 
return props; 



P9308 



47859 



Ver. 47859S5.doc 



27 



TBStringTokenizer st = 

new TBStringTokenizer ( testProps , delimeter) ; 
while ( St . hasMoreTokens ( ) ) { 
5 String token = st . nextToken ( ) ; 

try { 

String name = 

token . substring ( 0 , token . indexOf ( ) ) ; 
String value = token . substring ( token . indexOf ("=" ) 
10 + 1 , token . length() ) ; 

// index+1 to skip the sign 
props . set Property ( name , value ) ; 

} 

catch ( StringlndexOutOf BoundsException e) { 
15 e . printStackTrace ( ) ; 

System . out . println ( token + " property ignored"); 

} 

} 

return props ; 
20 } 

private String constructPropsString 
(Properties testProps) 

{ 

25 if (testProps == null) return 
String propsList = null; 

for (Enumeration e = testProps . propertyNames ( ); 
^ e . hasMoreElements ( ) ;) { 
30 String pName = (String) e . nextElement ( ) ; 

String prop = pName + + 

testProps . getProperty (pName ) ; 
if (propsList == null) 
propsList = prop; 
35 else 

propsList += ' + prop; 

} 

return propsList; 
40 } 

/** Searches all required files for this record, 
* including its own class 
*/ 

45 protected void listAllFiles ( ArrayList requiredFiles , 

TestRecord testRecord , 

String TestClassesDir ) { 
// adding required files and inner classes (of test 
// class and of required classes) 
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String required = 

testRecord . getValue ( TestRecord . RequiredClasses ) ; 
if ((required != null) && 

( required . trim( ). length ( ) != 0)) { 

5 

StringTokenizer ct = 

new StringTokenizer ( required, " , " ) ; 
while ( ct . hasMoreTokens ( ) ) { 
String reqPath, reqFile; 
10 String requiredFile = ct . nextToken (). replace 

('/'^ sep.charAt(O) ) ; 

//required class is in a different package from 
//test class 

15 int sepPos = requiredFile . last IndexOf ( sep ) ; 

if ( sepPos ! = - 1 ) { 

reqPath = requiredFile . substring ( 0 , sepPos); 
reqFile = 

requiredFile . substring ( requiredFile . last IndexOf ( sep ) 
20 + 1 ) ; 

} 

else { 

reqPath = " " ; 

reqFile = requiredFile; 

25 } 

requiredFiles . add ( requiredFile ) ; 

if ( requiredFile . ends With (". class ") ) 
//search inner classes of required class 
30 searchlnnerCl asses ( reqPath , 

reqFile . substring ( reqFile . last IndexOf ( sep) + 1 , 
reqFile . indexOf ( " . class " ) ) , 
requiredFiles, TestClassesDir ) ; 

} 

35 } 

//adding test class and it's inner classes 
String testClassPackage = 

testRecord. getValue ( TestRecord. TestPackage) . replace ( ' , 
40 sep . charAt ( 0 ) ) ; 

requiredFiles . add( testClassPackage + sep + 

testRecord . getValue ( TestRecord . TestClass ) + class" ) ; 
search Inner CI asses ( testClassPackage , 

testRecord . getValue ( TestRecord . TestClass ) , 
45 requiredFiles, TestClassesDir); 

} 

/** 

* Looks for inner classes for the specified class 
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*/ 

protected void searchlnnerClasses ( String testPackage, 
String testClass, 
ArrayList requiredFiles , 
5 String TestClassesDir ) { 
try { 

ArrayList innerClasses = null; 



10 // look for inner classes in package 

if ( ! hashOf InnerClasses . containsKey ( testPackage ) ) { 

// check for inner classes in new package and add them to 

//hash 

File f = null; 
15 if (TestClassesDir . length( ) > 0) f = new 

File ( TestClassesDir + sep + 
testPackage . replace ( ' . ' . sep . charAt ( 0 ) ) ) ; 
else f = new File ( testPackage ) ; 

20 String[] foundFiles = f.list(); 

//put found inner classes into the hashtable 
if (foundFiles != null) 

for ( int i=0 ; i < foundFiles . length ; i+-h) 
25 if (foundFiles[i] .endsWith( class" ) && 

( foundFiles [i] . indexOf ( '$')!=- 1 ) ) { 
if (innerClasses == null) 
innerClasses = new ArrayList(); 
innerClasses . add( foundFiles [ i ] ) ; 

30 } 

if (innerClasses == null) innerClasses = new 

ArrayList ( 0 ) ; 
hashOf InnerClasses . put ( testPackage , innerClasses ) ; 

} 

35 else 

innerClasses = (ArrayList) 

hashOf InnerClasses . get ( testPackage ) ; 
//look for class inner classes 
Iterator iter = innerClasses . iterator () ; 
40 while ( iter . hasNext ( ) ) { 

String currentlnner = (String) iter.next(); 
if ( currentlnner . startsWith( testClass ) && 
( currentlnner . indexOf ("$" ) != -1)) 
requiredFiles . add( testPackage -h sep + 
45 currentlnner) ; 

} 

} 

catch (Exception e) { 
e . printStackTrace ( ) ; 
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System . out . println ( "Failed to search inner classes for 
class: " + testClass + " in package: " + testPackage) ; 
System . out . println 

( "Absence of inner classes might cause test execution 
5 errors" ) ; 

} 
} 

protected String checkAndAddProperty ( String agentProps , 
10 String field. String value) { 

int fieldlndex = agentProps . indexOf ( field) ; 
if (fieldlndex != - 1 ) { 

String firstPart = agentProps . substring ( 0 , fieldlndex + 
field. length( ) +1); 
15 String secontPart = agentProps . substring( fieldlndex + 

field. length( ) +1 ) ; 
int newlinelndex = secontPart . indexOf (( int )' \n ') ; 
String newValueString = value + 

secontPart . substring(newlinelndex) + "\n"; 
20 agentProps = firstPart + newValueString; 

} 

return agentProps ; 

} 

} 

25 



30 
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